"Effective robustness" measures the extra out-of-distribution (OOD) robustness

However, sampling is not the only source of uncertainty. In practice, distributions change between locations and across time.

In safety-critical applications, machine learning models should generalize well under worst-case distribution shifts, that is, have a small robust risk.

One of the primary drivers of the success of machine learning methods in open-world perception settings, such ascomputer vision [19]and NLP [8],has been the ability ofhigh-capacity function approximators, suchasdeepneuralnetworks,tolearngeneralizable modelsfromlargeamountsof data.

Thisresult shows that visual grounding/feature attribution can be ineffective at resolving shortcuts.

These shifts are defined via parametric changes in the causal mechanisms of observed variables, where constraints on parameters yield a "robustness set" of plausible distributions and acorresponding worst-case loss overthe set.